Mining : Basic Concepts

This survey reviews a broad array of techniques that are becoming available to mine textual data. It presents initially a three function (data collection, data warehousing, data exploitation) text mining architecture consisting of a six step text mining process (source selection, text retrieval, information extraction, data storage, data mining, presentation). It then presents some of the most widely used data and text mining techniques, including clustering and classification methods (nearest neighbor, relational learning models, genetic algorithms) and dependency models (graph-theoretic link analysis, linear regression and decision trees, nonlinear regression and neural networks). The survey finally illustrates some of their potential by describing the Office of Naval Research text mining pilot program. In the first year of that program, existing metadata from commercial bibliographic databases was used. There is presently an unacceptably long delay between the development of key component technologies for textual data mining and the deployment of the integrated tools that S&T sponsors need. The first year of the ONR text mining pilot program represents an initial attempt to bridge that gap. Important lessons have been learned about the use of text mining for management of science and technology research, but much remains to be done.